Parallel Matrix Multiplication : 2 D and 3 D FLAME Working Note # 62 Martin Schatz

We describe an extension of the Scalable Universal Matrix Multiplication Algorithms (SUMMA) from 2D to 3D process grids; the underlying idea is to lower the communication volume through storing redundant copies of one or more matrices. While SUMMA was originally introduced for block-wise matrix distributions, so that most of its communication was within broadcasts, this paper focuses on element-wise matrix distributions, which lead to allgather-based algorithms. We begin by describing an allgather-based 2D SUMMA, describe its generalization to 3D process grids, and then discuss theoretical and experimental performance benefits of the new algorithms.